{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom Document Loader\n",
    "This guide will demonstrate how to write custom document loading and file parsing logic; specifically, we'll see how to:<br>\n",
    "本指南将演示如何编写自定义文档加载和文件解析逻辑;具体来说，我们将了解如何：\n",
    "\n",
    "- Create a standard document Loader by sub-classing from BaseLoader.<br>\n",
    "通过对 from BaseLoader 进行子类化来创建标准文档加载器。\n",
    "\n",
    "- Create a parser using BaseBlobParser and use it in conjunction with Blob and BlobLoaders. This is useful primarily when working with files.<br>\n",
    "使用 和 BaseBlobParser 结合使用 Blob BlobLoaders 来创建解析器。这主要在处理文件时很有用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many document loaders invovle parsing files. The difference between such loaders usually stems from how the file is parsed rather than how the file is loaded. For example, you can use open to read the binary content of either a PDF or a markdown file, but you need different parsing logic to convert that binary data into text.<br>\n",
    "许多文档加载器涉及解析文件。这种加载器之间的区别通常源于文件的解析方式，而不是文件的加载方式。例如，您可以使用 open 它来读取 PDF 或 Markdown 文件的二进制内容，但您需要不同的解析逻辑来将该二进制数据转换为文本。\n",
    "\n",
    "As a result, it can be helpful to decouple the parsing logic from the loading logic, which makes it easier to re-use a given parser regardless of how the data was loaded.<br>\n",
    "因此，将解析逻辑与加载逻辑分离会很有帮助，这样无论数据是如何加载的，都可以更轻松地重用给定的解析器。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standard Document Loader\n",
    "A document loader can be implemented by sub-classing from a BaseLoader which provides a standard interface for loading documents.<br>\n",
    "文档加载器可以通过从提供 BaseLoader 标准接口加载文档的子类来实现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import AsyncIterator, Iterator\n",
    "\n",
    "from langchain_core.document_loaders import BaseLoader\n",
    "from langchain_core.documents import Document\n",
    "\n",
    "\n",
    "class CustomDocumentLoader(BaseLoader):\n",
    "    \"\"\"An example document loader that reads a file line by line.\"\"\"\n",
    "\n",
    "    def __init__(self, file_path: str) -> None:\n",
    "        \"\"\"Initialize the loader with a file path.\n",
    "\n",
    "        Args:\n",
    "            file_path: The path to the file to load.\n",
    "        \"\"\"\n",
    "        self.file_path = file_path\n",
    "\n",
    "    def lazy_load(self) -> Iterator[Document]:  # <-- Does not take any arguments\n",
    "        \"\"\"A lazy loader that reads a file line by line.\n",
    "\n",
    "        When you're implementing lazy load methods, you should use a generator\n",
    "        to yield documents one by one.\n",
    "        \"\"\"\n",
    "        with open(self.file_path, encoding=\"utf-8\") as f:\n",
    "            line_number = 0\n",
    "            for line in f:\n",
    "                yield Document(\n",
    "                    page_content=line,\n",
    "                    metadata={\"line_number\": line_number, \"source\": self.file_path},\n",
    "                )\n",
    "                line_number += 1\n",
    "\n",
    "    # alazy_load is OPTIONAL.\n",
    "    # If you leave out the implementation, a default implementation which delegates to lazy_load will be used!\n",
    "    async def alazy_load(\n",
    "        self,\n",
    "    ) -> AsyncIterator[Document]:  # <-- Does not take any arguments\n",
    "        \"\"\"An async lazy loader that reads a file line by line.\"\"\"\n",
    "        # Requires aiofiles\n",
    "        # Install with `pip install aiofiles`\n",
    "        # https://github.com/Tinche/aiofiles\n",
    "        import aiofiles\n",
    "\n",
    "        async with aiofiles.open(self.file_path, encoding=\"utf-8\") as f:\n",
    "            line_number = 0\n",
    "            async for line in f:\n",
    "                yield Document(\n",
    "                    page_content=line,\n",
    "                    metadata={\"line_number\": line_number, \"source\": self.file_path},\n",
    "                )\n",
    "                line_number += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "<class 'langchain_core.documents.base.Document'>\n",
      "page_content='meow meow🐱 \\n' metadata={'line_number': 0, 'source': '../data/meow.txt'}\n",
      "\n",
      "<class 'langchain_core.documents.base.Document'>\n",
      "page_content=' meow meow🐱 \\n' metadata={'line_number': 1, 'source': '../data/meow.txt'}\n",
      "\n",
      "<class 'langchain_core.documents.base.Document'>\n",
      "page_content=' meow😻😻' metadata={'line_number': 2, 'source': '../data/meow.txt'}\n"
     ]
    }
   ],
   "source": [
    "file_path=\"../data/meow.txt\"\n",
    "with open(file_path, \"w\", encoding=\"utf-8\") as f:\n",
    "    quality_content = \"meow meow🐱 \\n meow meow🐱 \\n meow😻😻\"\n",
    "    f.write(quality_content)\n",
    "\n",
    "loader = CustomDocumentLoader(file_path)\n",
    "\n",
    "## Test out the load interface\n",
    "# Avoid using it for production code since eager loading assumes that all the content can fit into memory, which is not always the case, especially for enterprise data. :::\n",
    "# 避免将其用于生产代码，因为急切加载假定所有内容都可以放入内存中，但情况并非总是如此，尤其是对于企业数据。\n",
    "for doc in loader.load():\n",
    "    print()\n",
    "    print(type(doc))\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BaseBlobParser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.document_loaders import BaseBlobParser, Blob\n",
    "\n",
    "\n",
    "class MyParser(BaseBlobParser):\n",
    "    \"\"\"A simple parser that creates a document from each line.\"\"\"\n",
    "\n",
    "    def lazy_parse(self, blob: Blob) -> Iterator[Document]:\n",
    "        \"\"\"Parse a blob into a document line by line.\"\"\"\n",
    "        line_number = 0\n",
    "        with blob.as_bytes_io() as f:\n",
    "            for line in f:\n",
    "                line_number += 1\n",
    "                yield Document(\n",
    "                    page_content=line,\n",
    "                    metadata={\"line_number\": line_number, \"source\": blob.source},\n",
    "                )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='meow meow🐱 \\r\\n', metadata={'line_number': 1, 'source': '../data/meow.txt'}),\n",
       " Document(page_content=' meow meow🐱 \\r\\n', metadata={'line_number': 2, 'source': '../data/meow.txt'}),\n",
       " Document(page_content=' meow😻😻', metadata={'line_number': 3, 'source': '../data/meow.txt'})]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "blob = Blob.from_path(\"../data/meow.txt\")\n",
    "parser = MyParser()\n",
    "list(parser.lazy_parse(blob))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='some data from memory\\n', metadata={'line_number': 1, 'source': None}),\n",
       " Document(page_content='meow', metadata={'line_number': 2, 'source': None})]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "blob = Blob(data=b\"some data from memory\\nmeow\")\n",
    "list(parser.lazy_parse(blob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Blob Loaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "82344c8df43d4ef18ae81eadf19af593",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='meow meow🐱 \\r\\n' metadata={'line_number': 1, 'source': '..\\\\data\\\\meow.txt'}\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders.blob_loaders import FileSystemBlobLoader\n",
    "\n",
    "blob_loader = FileSystemBlobLoader(path=\"../data\", glob=\"*.*\", show_progress=True)\n",
    "\n",
    "parser = MyParser()\n",
    "for blob in blob_loader.yield_blobs():\n",
    "    for doc in parser.lazy_parse(blob):\n",
    "        print(doc)\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generic Loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "691def7d2da3439e9c194371a03dec71",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='meow meow🐱 \\r\\n' metadata={'line_number': 1, 'source': '..\\\\data\\\\meow.txt'}\n",
      "page_content=' meow meow🐱 \\r\\n' metadata={'line_number': 2, 'source': '..\\\\data\\\\meow.txt'}\n",
      "page_content=' meow😻😻' metadata={'line_number': 3, 'source': '..\\\\data\\\\meow.txt'}\n",
      "... output truncated for demo purposes\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders.generic import GenericLoader\n",
    "\n",
    "loader = GenericLoader.from_filesystem(\n",
    "    path=\"../data\", glob=\"*.*\", show_progress=True, parser=MyParser()\n",
    ")\n",
    "\n",
    "for idx, doc in enumerate(loader.lazy_load()):\n",
    "    if idx < 5:\n",
    "        print(doc)\n",
    "\n",
    "print(\"... output truncated for demo purposes\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a47c3587f55d48509e06a3fc02d6e0e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='meow meow🐱 \\r\\n' metadata={'line_number': 1, 'source': '..\\\\data\\\\meow.txt'}\n",
      "page_content=' meow meow🐱 \\r\\n' metadata={'line_number': 2, 'source': '..\\\\data\\\\meow.txt'}\n",
      "page_content=' meow😻😻' metadata={'line_number': 3, 'source': '..\\\\data\\\\meow.txt'}\n",
      "... output truncated for demo purposes\n"
     ]
    }
   ],
   "source": [
    "from typing import Any\n",
    "\n",
    "class MyCustomLoader(GenericLoader):\n",
    "    @staticmethod\n",
    "    def get_parser(**kwargs: Any) -> BaseBlobParser:\n",
    "        \"\"\"Override this method to associate a default parser with the class.\"\"\"\n",
    "        return MyParser()\n",
    "\n",
    "loader = MyCustomLoader.from_filesystem(path=\"../data\", glob=\"*.*\", show_progress=True)\n",
    "\n",
    "for idx, doc in enumerate(loader.lazy_load()):\n",
    "    if idx < 5:\n",
    "        print(doc)\n",
    "\n",
    "print(\"... output truncated for demo purposes\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain0_1",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
